64 research outputs found
A Neuro-Augmented Observer for Robust Fault Detection in Nonlinear Systems
A new fault detection method using neural-networks-augmented state observer for nonlinear systems is presented in this paper. The novelty of the approach is that instead of approximating the entire nonlinear system with neural network, we only approximate the unmodeled part that is left over after linearization, in which a radial basis function (RBF) neural network is adopted. Compared with conventional linear observer, the proposed observer structure provides more accurate estimation of the system state. The state estimation error is proved to asymptotically approach zero by the Lyapunov method. An aircraft system demonstrates the efficiency of the proposed fault detection scheme, simulation results of which show that the proposed RBF neural network-based observer scheme is effective and has a potential application in fault detection and identification (FDI) for nonlinear systems
A Comprehensive Study on Knowledge Graph Embedding over Relational Patterns Based on Rule Learning
Knowledge Graph Embedding (KGE) has proven to be an effective approach to
solving the Knowledge Graph Completion (KGC) task. Relational patterns which
refer to relations with specific semantics exhibiting graph patterns are an
important factor in the performance of KGE models. Though KGE models'
capabilities are analyzed over different relational patterns in theory and a
rough connection between better relational patterns modeling and better
performance of KGC has been built, a comprehensive quantitative analysis on KGE
models over relational patterns remains absent so it is uncertain how the
theoretical support of KGE to a relational pattern contributes to the
performance of triples associated to such a relational pattern. To address this
challenge, we evaluate the performance of 7 KGE models over 4 common relational
patterns on 2 benchmarks, then conduct an analysis in theory, entity frequency,
and part-to-whole three aspects and get some counterintuitive conclusions.
Finally, we introduce a training-free method Score-based Patterns Adaptation
(SPA) to enhance KGE models' performance over various relational patterns. This
approach is simple yet effective and can be applied to KGE models without
additional training. Our experimental results demonstrate that our method
generally enhances performance over specific relational patterns. Our source
code is available from GitHub at
https://github.com/zjukg/Comprehensive-Study-over-Relational-Patterns.Comment: This paper is accepted by ISWC 202
When Do Program-of-Thoughts Work for Reasoning?
The reasoning capabilities of Large Language Models (LLMs) play a pivotal
role in the realm of embodied artificial intelligence. Although there are
effective methods like program-of-thought prompting for LLMs which uses
programming language to tackle complex reasoning tasks, the specific impact of
code data on the improvement of reasoning capabilities remains under-explored.
To address this gap, we propose complexity-impacted reasoning score (CIRS),
which combines structural and logical attributes, to measure the correlation
between code and reasoning abilities. Specifically, we use the abstract syntax
tree to encode the structural information and calculate logical complexity by
considering the difficulty and the cyclomatic complexity. Through an empirical
analysis, we find not all code data of complexity can be learned or understood
by LLMs. Optimal level of complexity is critical to the improvement of
reasoning abilities by program-aided prompting. Then we design an
auto-synthesizing and stratifying algorithm, and apply it to instruction
generation for mathematical reasoning and code data filtering for code
generation tasks. Extensive results demonstrates the effectiveness of our
proposed approach. Code will be integrated into the EasyInstruct framework at
https://github.com/zjunlp/EasyInstruct.Comment: Work in progres
Disentangled Contrastive Learning for Learning Robust Textual Representations
Although the self-supervised pre-training of transformer models has resulted
in the revolutionizing of natural language processing (NLP) applications and
the achievement of state-of-the-art results with regard to various benchmarks,
this process is still vulnerable to small and imperceptible permutations
originating from legitimate inputs. Intuitively, the representations should be
similar in the feature space with subtle input permutations, while large
variations occur with different meanings. This motivates us to investigate the
learning of robust textual representation in a contrastive manner. However, it
is non-trivial to obtain opposing semantic instances for textual samples. In
this study, we propose a disentangled contrastive learning method that
separately optimizes the uniformity and alignment of representations without
negative sampling. Specifically, we introduce the concept of momentum
representation consistency to align features and leverage power normalization
while conforming the uniformity. Our experimental results for the NLP
benchmarks demonstrate that our approach can obtain better results compared
with the baselines, as well as achieve promising improvements with invariance
tests and adversarial attacks. The code is available in
https://github.com/zjunlp/DCL.Comment: Work in progres
CodeKGC: Code Language Model for Generative Knowledge Graph Construction
Current generative knowledge graph construction approaches usually fail to
capture structural knowledge by simply flattening natural language into
serialized texts or a specification language. However, large generative
language model trained on structured data such as code has demonstrated
impressive capability in understanding natural language for structural
prediction and reasoning tasks. Intuitively, we address the task of generative
knowledge graph construction with code language model: given a code-format
natural language input, the target is to generate triples which can be
represented as code completion tasks. Specifically, we develop schema-aware
prompts that effectively utilize the semantic structure within the knowledge
graph. As code inherently possesses structure, such as class and function
definitions, it serves as a useful model for prior semantic structural
knowledge. Furthermore, we employ a rationale-enhanced generation method to
boost the performance. Rationales provide intermediate steps, thereby improving
knowledge extraction abilities. Experimental results indicate that the proposed
approach can obtain better performance on benchmark datasets compared with
baselines. Code and datasets are available in
https://github.com/zjunlp/DeepKE/tree/main/example/llm.Comment: Work in progres
On Robustness and Bias Analysis of BERT-based Relation Extraction
Fine-tuning pre-trained models have achieved impressive performance on
standard natural language processing benchmarks. However, the resultant model
generalizability remains poorly understood. We do not know, for example, how
excellent performance can lead to the perfection of generalization models. In
this study, we analyze a fine-tuned BERT model from different perspectives
using relation extraction. We also characterize the differences in
generalization techniques according to our proposed improvements. From
empirical experimentation, we find that BERT suffers a bottleneck in terms of
robustness by way of randomizations, adversarial and counterfactual tests, and
biases (i.e., selection and semantic). These findings highlight opportunities
for future improvements. Our open-sourced testbed DiagnoseRE is available in
\url{https://github.com/zjunlp/DiagnoseRE}.Comment: work in progres
OceanGPT: A Large Language Model for Ocean Science Tasks
Ocean science, which delves into the oceans that are reservoirs of life and
biodiversity, is of great significance given that oceans cover over 70% of our
planet's surface. Recently, advances in Large Language Models (LLMs) have
transformed the paradigm in science. Despite the success in other domains,
current LLMs often fall short in catering to the needs of domain experts like
oceanographers, and the potential of LLMs for ocean science is under-explored.
The intrinsic reason may be the immense and intricate nature of ocean data as
well as the necessity for higher granularity and richness in knowledge. To
alleviate these issues, we introduce OceanGPT, the first-ever LLM in the ocean
domain, which is expert in various ocean science tasks. We propose DoInstruct,
a novel framework to automatically obtain a large volume of ocean domain
instruction data, which generates instructions based on multi-agent
collaboration. Additionally, we construct the first oceanography benchmark,
OceanBench, to evaluate the capabilities of LLMs in the ocean domain. Though
comprehensive experiments, OceanGPT not only shows a higher level of knowledge
expertise for oceans science tasks but also gains preliminary embodied
intelligence capabilities in ocean technology. Codes, data and checkpoints will
soon be available at https://github.com/zjunlp/KnowLM.Comment: Work in progress. Project Website:
https://zjunlp.github.io/project/OceanGPT
Learning to Ask for Data-Efficient Event Argument Extraction
Event argument extraction (EAE) is an important task for information
extraction to discover specific argument roles. In this study, we cast EAE as a
question-based cloze task and empirically analyze fixed discrete token template
performance. As generating human-annotated question templates is often
time-consuming and labor-intensive, we further propose a novel approach called
"Learning to Ask," which can learn optimized question templates for EAE without
human annotations. Experiments using the ACE-2005 dataset demonstrate that our
method based on optimized questions achieves state-of-the-art performance in
both the few-shot and supervised settings.Comment: work in progres
Normal vs. Adversarial: Salience-based Analysis of Adversarial Samples for Relation Extraction
Recent neural-based relation extraction approaches, though achieving
promising improvement on benchmark datasets, have reported their vulnerability
towards adversarial attacks. Thus far, efforts mostly focused on generating
adversarial samples or defending adversarial attacks, but little is known about
the difference between normal and adversarial samples. In this work, we take
the first step to leverage the salience-based method to analyze those
adversarial samples. We observe that salience tokens have a direct correlation
with adversarial perturbations. We further find the adversarial perturbations
are either those tokens not existing in the training set or superficial cues
associated with relation labels. To some extent, our approach unveils the
characters against adversarial samples. We release an open-source testbed,
"DiagnoseAdv" in https://github.com/zjunlp/DiagnoseAdv.Comment: IJCKG 202
Contrastive Demonstration Tuning for Pre-trained Language Models
Pretrained language models can be effectively stimulated by textual prompts
or demonstrations, especially in low-data scenarios. Recent works have focused
on automatically searching discrete or continuous prompts or optimized
verbalizers, yet studies for the demonstration are still limited. Concretely,
the demonstration examples are crucial for an excellent final performance of
prompt-tuning. In this paper, we propose a novel pluggable, extensible, and
efficient approach named contrastive demonstration tuning, which is free of
demonstration sampling. Furthermore, the proposed approach can be: (i) Plugged
to any previous prompt-tuning approaches; (ii) Extended to widespread
classification tasks with a large number of categories. Experimental results on
16 datasets illustrate that our method integrated with previous approaches
LM-BFF and P-tuning can yield better performance. Code is available in
https://github.com/zjunlp/PromptKG/tree/main/research/Demo-Tuning.Comment: Work in progres
- …